Medical Imaging Simulators
Application of ultrasound requires a high level of expertise both in manipulating imaging devices and also in analyzing and interpreting resulting images, for instance in the medical field for accurate diagnosis and intervention guidance. Learning the proper execution of this modality thus requires a lengthy training period for ultrasound specialists.
To facilitate the training of medical students and doctors, advanced medical procedure simulators may be used such as the one described in U.S. Pat. No. 8,992,230. Such simulators may be based on a virtual reality (“VR”) and/or a mixed or augmented reality (“AR”) simulation apparatus, by which the physician may experiment within a medical procedure scenario. The VR/AR system may compute and display a visual VR/AR model of anatomical structures in accordance with physician gestures and actions to provide various feedback, such as visual feedback. In a VR system, an entire image may be simulated for display to a user, and in an AR system, a simulated image may be overlaid or otherwise incorporated with an actual image for display to a user. Various patient models with different pathologies can be selected. Therefore, natural variations as encountered over the years by practicing medical staff can be simulated for a user over a compressed period of time for training purposes.
Ultrasound Imaging Simulation
Early ultrasound simulation solutions have been developed based on interpolative ultrasound simulation, such as for instance the method developed by O. Goksel and S. E. Salcudean as described in “B-mode ultrasound image simulation in deformable 3-D medium”, IEEE Trans Medical Imaging, 28(11):1657-69, November 2009 [Goksel 2009]. Interpolative approaches can generally generate realistic images, but only in the absence of directional image artifacts and for images from limited fields of view. In order to handle different field-of-views and to better simulate certain artifacts, as required by certain ultrasound applications such as abdominal ultrasound, other approaches are needed.
Generative Simulation
Generative simulation, such as wave-based or ray-based ultrasound simulation, aims at emulating the ultrasonic signal that would be registered by a transducer position/orientation, using a combination of analytical and numerical solutions in real-time. However, simulating all possible ultrasound-tissue interaction phenomena is still an unsolved theoretical problem. For instance, the ultrasound texture (speckles) is a result of constructive and destructive interference of ultrasonic waves mainly scattered by sub-wavelength particles, such as cell nuclei, other organelles, etc. However, no known method can observe a sufficiently large tissue region (40-150 mm for OB/GYN ultrasound examination) in such fine detail with cellular structures.
An ultrasound transducer consists of several (e.g., 128 or 512) piezoelectric elements that can both transmit and receive ultra-sound, through conversion between electricity and vibration. The transmitted pressure wave (ultrasound) then interacts with anatomical structures with different acoustic impedances, where any reflected signal is again digitized by the same elements to be used for generating an image of the underneath tissue. Ultrasound (US) interaction with tissue happens in two different ways:                On the one hand, structures much smaller than the US wavelength (≈300 νm) absorb US energy and re-emit (scatter) it omni-directionally as point sources, such as cell nuclei, large proteins, etc. This interference pattern is indeed the source of the typical noisy texture of ultra-sound, called the speckle.        On the other hand, any macroscopic interface of impedance difference causes the US wave to be both reflected and refracted given its incidence angle. Accordingly, US may present both wave-like and ray-like properties, in a way similarly to light. Although the wave properties can be simulated in the entire domain, e.g. using finite difference methods, since the main ultrasound energy is focused in a certain direction, its wave-front can also be modeled as ray propagation in tissue.        
A common model for simulating US interaction with sub-wavelength particles is to assume that tissue is populated with many (but countable number of) scatterers. These can then have varying scattering amplitudes, and act as spherical sources of scattering for incident ultrasound signal. Prior art performance-optimized have been developed by T. Varslot in Varslot T., Taraldaen G.: “Computer simulation of forward wave propagation in soft tissue”, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 52, 9 (September 2005), 1473-1482 or in Varslot T., “Forward propagation of acoustic pressure pulses in 3d soft biological tissue”, Modeling, Identification and Control (2006) and GPU-accelerated in Karamalis A. et al, “Fast ultrasound image simulation using the Westervelt equation”, Proceedings of MICCAI (2010), pp. 243-250, but this wave simulations for ultrasound are still not fast enough to handle the complexity of clinical ultrasound in real-time. The computational complexity of such wave simulations restricts their use to offline simulation, as used in specific applications e.g., for transducer design and image processing algorithm development and validation. In the context of training simulators however these approaches are not applicable, as the generation of US images at interactive rates is essential and hence of high interest. A viable approximation of the full wave model is the convolution model for ultrasound speckle where the received intensity of the ultrasound is obtained by convolving scatterers with the point spread function (PSF) of the incident ultrasound energy at that location. Assuming a PSF separable in 3 axes and a discrete grid approximation, fast separable convolution of scatters was shown to simulate speckles interactively on modern GPUs (GAO H. et al. in “A fast convolution-based methodology to simulate 2-D/3-D cardiac ultrasound images”, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 56, 2 (February 2009), 404-409 [GCC*09]).
However, scatterer-based methods do not inherently take into account reflection and distortion of the ultrasound beam at the interfaces between large-scale structures, the source of artifacts such as acoustic shadowing, ghosting, and multiple reflections. These are indeed important cues for differential diagnosis of pathology, and thus should be part of a realistic simulation. In ultrasound beamforming, incoherent (out-of-phase) ultrasound waves cancel out contributions of each other, creating a focused beam of ultrasound. The interaction of beamformed ultrasound with macroscopic structures like bones or organ surfaces can hence be well captured by ray-tracing techniques known from computer graphics. Prior art methods thus combine separable convolution for wave-like interference with fast surface-based ray-tracing and in particular make use of dedicated GPU-based rendering frameworks of mesh representations such as NVIDIA OptiX, as described for instance by Benny Bürger, Sascha Bettinghausen, Matthias Radle, and Jürgen Hesser, in “Real-time GPU-based ultrasound simulation using deformable mesh models”, IEEE Transactions on Medical Imaging, 32(3):609-618, 2013 [BBRH13] and Salehi et al. “Patient-specific 3D ultrasound simulation based on convolutional ray-tracing and appearance optimization”, Proceedings of MICCAI (2015), pp. 510-518 [SAP*15]. These prior art surface-based ray-tracing methods utilize a recursive ray-tracing scheme: whenever a ray terminates at a surface, a new refraction ray and a new reflection ray are cast according to Snell's law in a binary tree structure, until the contribution of a ray is smaller than a threshold. Such deterministic schemes make the assumption that large-scale structures behave like perfect mirrors with infinitely sharp reflections and refractions. This may be true for numerical phantoms and artificial tissue-mimicking material; however, in actual anatomy perfect reflections and refractions are never the case, hence these methods often produce artificial, toy-like images not comparable to actual patient ultrasound. For instance, the state-of-the-art methods [BBRH13,SAP*15] evaluate a diffuse reflection term only locally (similar to specular reflections in computer graphics) but do not take non-specular indirect contributions into account, hence suffering from “hard” shadows, sharp reflections, and too crisp structure borders—atypical in real US imaging. [SAP*15] further attempts to address these issues with a post-processing step to make images visually closer to expected; this, nevertheless, does not deal with the main problem of the wrong initial assumptions and can only partially improve the realism, while requiring additional post-processing at the expense of the overall computation efficiency and complexity.
We also observed that for reflections from rough (imperfect) surfaces, contributions from many directions have to be considered. In a framework of deterministic ray-tracing, it may be possible to cast multiple rays at each intersection point of the parent ray in order to integrate over their individual contributions. One major limitation of these deterministic ray-tracing approaches is that deeper recursion levels require summing over an exponentially-growing number of rays, although these contribute less and less to the final image. Therefore, these algorithms quickly become computationally inefficient. Furthermore, such method exhibits poor parallelism, since multiple rays on subsequent levels are dependent on the single parent rays. Introduced in the late seventies, this deterministic method has long been shown to be inferior to stochastic techniques for natural scenes, such as Monte-Carlo path tracing methods. Surprisingly, however, stochastic techniques have not been investigated in the literature for the purpose of ultrasound image simulation. Recently, Monte-Carlo techniques have been utilized for other uses of medical ultrasound, such as focused ultrasound therapy in Koskela J. et al., “Stochastic ray tracing for simulation of high intensity focal ultrasound therapy”, The Journal of the Acoustical Society of America 136, 3 (2014), 1430-1440 [KVdG*14], where they have shown excellent performance and inherent parallelism. However, the purpose of the latter application is not to emulate the image formation process of ultrasound machines, and since therapy ultrasound frequencies cannot be used for image generation, these methods cannot directly be used to simulate radio-frequency or B-mode images. In particular the specific issues of generating realistic ultrasound images with the proper reflection and refraction for different anatomy component properties has not been investigated in the state of the art works for ultrasound simulation.
There is therefore a need for improved ray-tracing methods and systems to model and compute (diffuse) multiple reflections, which can be parallelized on a GPU architecture, to facilitate real-time, realistic, interactive ultrasound imaging simulation.